Systems and methods for audio medical instrument patient measurements

ABSTRACT

Presented are systems and methods for the accurate acquisition of medical measurement data of a body part of patient. To assist in acquiring accurate medical measurement data, an automated diagnostic and treatment system provides instructions to the patient to allow the patient to precisely position a medical instrument in proximity to a target spot of a body part of patient. For a stethoscope examination, the steps may include utilizing object tracking to determine if the patient has moved the stethoscope to a recording site; utilizing DSP processing to confirm that the stethoscope is in operation, utilizing DSP processing to generate a pre-processed audio sample from a recorded audio signal; using machine learning (ML) to determine if a signal of interest (SOI) is present in the pre-processed sample. If SoI is present, using ML to evaluate characteristics in the signal which indicate the presence of abnormalities in the organ being measured.

BACKGROUND Technical Field

The present disclosure relates to computer-assisted health care, andmore particularly, to systems and methods for providing acomputer-assisted medical diagnostic architecture in which a patient isable to perform certain medical measurements with the assistance of amedical kiosk or medical camera system.

Background of the Invention

One skilled in the art will recognize the importance of addressing theever-increasing cost of providing consistent, high-quality medical careto patients. Governmental agencies and insurance companies areattempting to find solutions that reduce the cost of medical carewithout significantly reducing the quality of medical examinationsprovided by doctors and nurses. Through consistent regulation changes,electronic health record changes and pressure from payers, both healthcare facilities and providers are looking for ways to make patientintake, triage, diagnosis, treatment, electronic health record dataentry, treatment, billing, and patient follow-up activity moreefficient, provide better patient experience, and increase the doctor topatient throughput per hour, while simultaneously reducing cost.

The desire to increase access to health care providers, a pressing needto reduce health care costs in developed countries and the goal ofmaking health care available to a larger population in less developedcountries have fueled the idea of telemedicine. In most cases, however,video or audio conferencing with a doctor does not provide sufficientpatient-physician interaction that is necessary to allow for a propermedical diagnosis to efficiently serve patients.

What is needed are systems and methods that ensure reliable remote orlocal medical patient intake, triage, diagnosis, treatment, electronichealth record data entry/management, treatment, billing and patientfollow-up activity so that physicians can allocate patient time moreefficiently and, in some instances, allow individuals to manage theirown health, thereby, reducing health care costs.

BRIEF DESCRIPTION OF THE DRAWINGS

References will be made to embodiments of the invention, examples ofwhich may be illustrated in the accompanying figures. These figures areintended to be illustrative, not limiting. Although the invention isgenerally described in the context of these embodiments, it should beunderstood that it is not intended to limit the scope of the inventionto these particular embodiments.

FIG. 1 illustrates an exemplary medical diagnostic system according toembodiments of the present disclosure.

FIG. 2 illustrates an exemplary medical instrument equipment systemaccording to embodiments of the present disclosure.

FIG. 3A illustrates an exemplary system for measuring a patient's bodyparts utilizing audio medical instruments and deep learning systemsaccording to embodiments of the present disclosure.

FIG. 3B illustrates another exemplary system for measuring a patient'sbody parts utilizing cameras and sensors according to embodiments of thepresent disclosure

FIGS. 4A, 4B, 4C, and 4D illustrates an exemplary method for identifyinga patient's body parts and body recording sites according to embodimentsof the present disclosure.

FIG. 4E illustrates exemplary method for the identification of bodyparts of a patient and the positioning of a stethoscope for recordingaudio measurement of the body according to embodiments of the presentdisclosure.

FIGS. 5A and 5B are flowcharts of exemplary methods for making accuratemedical instrument patient audio measurements according to embodimentsof the present disclosure.

FIG. 5C is a flowchart of an illustrative exemplary method forpositioning a stethoscope within a recording threshold to recordingsites of body parts of the patient according to embodiments of thepresent disclosure

FIG. 6 depicts a simplified block diagram of a computingdevice/information handling system according to embodiments of thepresent disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the following description, for purposes of explanation, specificdetails are set forth in order to provide an understanding of thedisclosure. It will be apparent, however, to one skilled in the art thatthe disclosure can be practiced without these details. Furthermore, oneskilled in the art will recognize that embodiments of the presentdisclosure, described below, may be implemented in a variety of ways,such as a process, an apparatus, a system, a device, or a method on atangible computer-readable medium.

Elements/components shown in diagrams are illustrative of exemplaryembodiments of the disclosure and are meant to avoid obscuring thedisclosure. It shall also be understood that throughout this discussionthat components may be described as separate functional units, which maycomprise sub-units, but those skilled in the art will recognize thatvarious components, or portions thereof, may be divided into separatecomponents or may be integrated together, including integrated within asingle system or component. It should be noted that functions oroperations discussed herein may be implemented as components/elements.Components/elements may be implemented in software, hardware, or acombination thereof.

Furthermore, connections between components or systems within thefigures are not intended to be limited to direct connections. Rather,data between these components may be modified, re-formatted, orotherwise changed by intermediary components. Also, additional or fewerconnections may be used. Also, additional or fewer connections may beused. It shall also be noted that the terms “coupled” “connected” or“communicatively coupled” shall be understood to include directconnections, indirect connections through one or more intermediarydevices, and wireless connections.

Reference in the specification to “one embodiment,” “preferredembodiment,” “an embodiment,” or “embodiments” means that a particularfeature, structure, characteristic, or function described in connectionwith the embodiment is included in at least one embodiment of thedisclosure and may be in more than one embodiment. The appearances ofthe phrases “in one embodiment,” “in an embodiment,” or “in embodiments”in various places in the specification are not necessarily all referringto the same embodiment or embodiments. The terms “include,” “including,”“comprise,” and “comprising” shall be understood to be open terms andany lists that follow are examples and not meant to be limited to thelisted items. Any headings used herein are for organizational purposesonly and shall not be used to limit the scope of the description or theclaims.

Furthermore, the use of certain terms in various places in thespecification is for illustration and should not be construed aslimiting. A service, function, or resource is not limited to a singleservice, function, or resource; usage of these terms may refer to agrouping of related services, functions, or resources, which may bedistributed or aggregated.

In this document, the term “sensor” refers to a device capable ofacquiring information related to any type of physiological condition oractivity (e.g., a biometric diagnostic sensor); physical data (e.g., aweight); and environmental information (e.g., ambient temperaturesensor), including hardware-specific information. The term “position”refers to spatial and temporal data (e.g. orientation and motioninformation). The position data may be, for example, but withoutlimitations, 2-dimensional (2D), 2.5-dimensional (2.5D), or true3-dimensional (3D) data representation. “Doctor” refers to any healthcare professional, health care provider, physician, or person directedby a physician. “Patient” is any user who uses the systems and methodsof the present invention, e.g., a person being examined or anyoneassisting such person. The term illness may be used interchangeably withthe term diagnosis. As used herein, “answer” or “question” refers to oneor more of 1) an answer to a question, 2) a measurement or measurementrequest (e.g., a measurement performed by a “patient”), and 3) a symptom(e.g., a symptom selected by a “patient”).

FIG. 1 illustrates an exemplary medical diagnostic system according toembodiments of the present disclosure. Diagnostic system 100 comprisesautomated diagnostic and treatment system 102, patient interface station106, doctor interface station 104, deep learning system 105,sensors/cameras 109 and medical instrument equipment 108. Deep learningsystem 105 may include, for example, but without limitations, deepconvolution neural network (DCNN), Multi-layer preceptrons (MLP), FullyConvolutional Networks (FCNs), Capsule Networks, and artificial neuralnetworks (A-NNs). Both patient interface station 106 and doctorinterface station 104 may be implemented into any tablet, computer,mobile device, or other electronic device. Deep learning system 105utilizes virtual, augment, or mixed reality to assist the automateddiagnostic and treatment system 102 to obtain improved medicalmeasurements by the medical instrument equipment 108. Deep learning maybe a name referring to the use of “stacked neural networks”; that is,networks composed of several layers. The layers are made of nodes, wherea node is just a place where computation occurs and is loosely patternedon a neuron in the human brain. The neuron in the brain fires when itencounters sufficient stimuli. Deep learning system 105 may be used incomputer vision and have also have been applied to acoustic modeling forautomatic speech recognition (ASR). Automated diagnostic and treatmentsystem 102 may be referred to as an automated diagnostic system, orsimply diagnostic system.

Patient interface station 106 comprises patient interface applicationmodule 107 and is coupled to sensors/cameras 109. In turn,sensors/cameras 109 may monitor and may capture images of thepatient/assistant. Patient interface station 106 receives from theautomated diagnostic and treatment system 102 system instruction and/orrelayed physicians instruction, diagnostic feedback, results 126.Patient interface station 106 sends to the automated diagnostic andtreatment system 102 secure raw or processed patient-related data 128.Patient interface station 106 also provides interfaces betweensensors/cameras 109, located in the patient interface station 106, andthe patient/assistant. Patient interface station 106 also providesinterfaces to medical instrument equipment 108. Medical instrumentequipment 108 may be, for example, but without limitations, astethoscope, ophthalmoscope, intraoral camera, auriscope, or otoscope.

Medical instrument equipment 108 is designed to collect mainlydiagnostic patient data, and may comprise one or more diagnosticdevices, for example, in a home diagnostic medical kit that generatesdiagnostic data based on physical and non-physical characteristics of apatient. It is noted that diagnostic system 100 may comprise additionalsensors and devices that, in operation, collect, process, or transmitcharacteristic information about the patient, medical instrument usage,orientation, environmental parameters such as ambient temperature,humidity, location, and other useful information that may be used toaccomplish the objectives of the present invention.

In operation, a patient may enter patient-related data, such as healthhistory, patient characteristics, symptoms, health concerns, medicalinstrument measured diagnostic data, images, and sound patterns, orother relevant information into patient interface station 106. Thepatient may use any means of communication, such as voice control, toenter data, e.g., in the form of a questionnaire. Patient interfacestation 106 may provide the data raw or in processed form to automateddiagnostic and treatment system 102, e.g., via a secure communication.

In embodiments, the patient may be prompted, e.g., by a softwareapplication, to answer questions intended to aid in the diagnosis of oneor more medical conditions. The software application may provideguidance by describing how to use medical instrument equipment 108 toadminister a diagnostic test or how to make diagnostic measurements forany particular device that may be part of medical instrument equipment108 so as to facilitate accurate measurements of patient diagnosticdata.

In embodiments, the patient may use medical instrument equipment 108 tocreate a patient health profile that serves as a baseline profile.Gathered patient-related data may be securely stored in database 103 ora secure remote server (not shown) coupled to automated diagnostic andtreatment system 102. In embodiments, automated diagnostic and treatmentsystem 102, supported by and deep learning system 105, enablesinteraction between a patient and a remotely located health careprofessional, who may provide instructions to the patient, e.g., bycommunicating via the software application. A doctor may log into acloud-based system (not shown) to access patient-related data via doctorinterface station 104. The doctor interface station 104 comprises adoctor interface communication module 130. In embodiments, automateddiagnostic and treatment system 102 presents automated diagnosticsuggestions to a doctor, who may verify or modify the suggestedinformation. Automated diagnostic and treatment system 102 is coupled todoctor interface station 104. Automated diagnostic and treatment system102 may communicate to doctor interface station 104 alerts, if needed,risk profile, and data integrity 120.

In embodiments, based on one more patient questionnaires, data gatheredby medical instrument equipment 108, patient feedback, and historicdiagnostic information, the patient may be provided with instructions,feedback, results 122, and other information pertinent to the patient'shealth. In embodiments, the doctor may select an illness based onautomated diagnostic system suggestions and/or follow a sequence ofinstructions, feedback, and/or results 122 may be adjusted based ondecision vectors associated with a medical database. In embodiments,medical instrument equipment 108 uses the decision vectors to generate adiagnostic result, e.g., in response to patient answers and/ormeasurements of the patient's vital signs.

In embodiments, medical instrument equipment 108 comprises a number ofsensors, such as accelerometers, gyroscopes, pressure sensors, cameras,bolometers, altimeters, IR LEDs, and proximity sensors that may becoupled to one or more medical devices, e.g., a thermometer, to assistin performing diagnostic measurements and/or monitor a patient's use ofmedical instrument equipment 108 for accuracy. A camera, bolometer, orother spectrum imaging device (e.g. radar), in addition to takingpictures of the patient, may use image or facial recognition softwareand machine vision to recognize body parts, items, and actions to aidthe patient in locating suitable positions for taking a measurement onthe patient's body, e.g., by identifying any part of the patient's bodyas a reference.

Examples of the types of diagnostic data that medical instrumentequipment 108 may generate comprise body temperature, blood pressure,images, sound, heart rate, blood oxygen level, motion, ultrasound,pressure or gas analysis, continuous positive airway pressure,electrocardiogram, electroencephalogram, electrocardiography, BMI,muscle mass, blood, urine, and any other patient-related data 128. Inembodiments, patient-related data 128 may be derived from a non-surgicalwearable or implantable monitoring device that gathers sample data.

In embodiments, an IR LED, proximity beacon, or other identifiablemarker (not shown) may be attached to medical instrument equipment 108to track the position and placement of medical instrument equipment 108.In embodiments, a camera, bolometer, or other spectrum imaging deviceuses the identifiable marker as a control tool to aid the camera or thepatient in determining the position of medical instrument equipment 108.

In embodiments, a deep learning system 105, which may include machinevision software, may be used to track and overlay or superimpose, e.g.,on a screen, the position of the identifiable marker e.g., IR LED, heatsource, or reflective material with a desired target location at whichthe patient should place medical instrument equipment 108, thereby,aiding the patient to properly place or align a sensor and ensureaccurate and reliable readings. Once medical instrument equipment 108,e.g., a stethoscope is placed at the desired target location on apatient's torso, the patient may be prompted by optical, haptic orvisual cues to breath according to instructions or perform other actionsto facilitate medical measurements and to start a measurement.

In embodiments, one or more sensors that may be attached to medicalinstrument equipment 108 monitor the placement and usage of medicalinstrument equipment 108 by periodically or continuously recording dataand comparing measured data, such as location, movement, rotation andangles, to an expected data model and/or an error threshold to ensuremeasurement accuracy. A patient may be instructed to adjust an angle,location, rotation or motion of medical instrument equipment 108, e.g.,to adjust its state and, thus, avoid low-accuracy or faulty measurementreadings. In embodiments, sensors attached or tracking medicalinstrument equipment 108 may generate sensor data and patientinteraction activity data that may be compared, for example, against anidealized patient medical instrument equipment usage sensor model datato create an equipment usage accuracy score. The patient medicalinstrument equipment measured medical data may also be compared withidealized device measurement data expected from medical instrumentequipment 108 to create a device accuracy score. Recording bioelectricsignals may include, for example, but without limitations,electroencephalography, electrocardiography, electrooculography,electromyography and electrogastrography.

Feedback from medical instrument equipment 108 (e.g., sensors,proximity, camera, etc.) and actual device measurement data may be usedto instruct the patient to properly align medical instrument equipment108 during a measurement. In embodiments, medical instrument equipmenttype and sensor system monitoring of medical instrument equipment 108patient interaction may be used to create a device usage accuracy scorefor use in a medical diagnosis algorithm. Similarly, patient medicalinstrument equipment measured medical data may be used to create ameasurement accuracy score for use by the medical diagnostic algorithm.In embodiments, medical instrument equipment 108 may also be fitted withhaptic/tactile feedback sensors which may augment audio or screen-basedguidance instructions for taking measurement. These haptic feedbackdevices may include, but are not limited to, vibration motors (similarto those found in smart phones) fitted around the medical instrumentequipment to give tactile instructions (e.g. vibration on the left sideof the medical device to instruct the patient to move the medical deviceleft), or flywheels to give kinesthetic feedback such as resistingmovement in undesirable directions, and aiding movement in the correctdirection (similar to certain virtual reality peripherals).

In embodiments, deep learning system 105, which may include machinevision software, may be used to show on a display an animation thatmimics a patient's movements and provides detailed interactiveinstructions and real-time feedback to the patient. This aids thepatient in correctly positioning and operating medical instrumentequipment 108 relative to the patient's body so as to ensure a highlevel of accuracy when using medical instrument equipment 108 isoperated. In embodiments, the display may be a traditional computerscreen. In other embodiments, the display may be augmented/virtual/mixedreality hardware, where patients may interact with the kiosk in acompletely virtual environment while wearing AR/VR goggles.

In embodiments, once automated diagnostic and treatment system 102detects unexpected data, e.g., data representing an unwanted movement,location, measurement data, etc., a validation process comprising acalculation of a trustworthiness score or reliability factor isinitiated in order to gauge the measurement accuracy. Once the accuracyof the measured data falls below a desired level, the patient may beasked to either repeat a measurement or request assistance by anassistant, who may answer questions, e.g., remotely via an applicationto help with proper equipment usage, or alert a nearby person to assistwith using medical instrument equipment 108. The validation process, mayalso instruct a patient to answer additional questions, and may comprisecalculating the measurement accuracy score based on a measurement orre-measurement.

In embodiments, upon request 124, automated diagnostic and treatmentsystem 102 may enable a patient-doctor interaction by granting thepatient and doctor access to diagnostic system 100. The patient mayenter data, take measurements, and submit images and audio files or anyother information to the application or web portal. The doctor mayaccess that information, for example, to review a diagnosis generated byautomated diagnostic and treatment system 102, and generate, confirm, ormodify instructions for the patient. Patient-doctor interaction, whilenot required for diagnostic and treatment, if used, may occur in person,real-time via an audio/video application, or by any other means ofcommunication.

In embodiments, automated diagnostic and treatment system 102 mayutilize images generated from a diagnostic examination of mouth, throat,eyes, ears, skin, extremities, surface abnormalities, internal imagingsources, and other suitable images and/or audio data generated fromdiagnostic examination of heart, lungs, abdomen, chest, joint motion,voice, and any other audio data sources. Automated diagnostic andtreatment system 102 may further utilize patient lab tests, medicalimages, or any other medical data. In embodiments, automated diagnosticand treatment system 102 enables medical examination of the patient, forexample, using medical devices, e.g., ultrasound, in medical instrumentequipment 108 to detect sprains, contusions, or fractures, andautomatically provide diagnostic recommendations regarding a medicalcondition of the patient. One skilled in the art may recognize thatother methods and systems may be utilized to enable medical examinationsof the patient.

In embodiments, diagnosis comprises the use of medical database decisionvectors that are at least partially based on the patient's self-measured(or assistant-measured) vitals or other measured medical data, theaccuracy score of a measurement dataset, a usage accuracy score of asensor attached to medical instrument equipment 108, a regional illnesstrend, and information used in generally accepted medical knowledgeevaluations steps. The decision vectors and associated algorithms, whichmay be installed in automated diagnostic and treatment system 102, mayutilize one or more-dimensional data, patient history, patientquestionnaire feedback, and pattern recognition or pattern matching forclassification using images and audio data. In embodiments, a medicaldevice usage accuracy score generator (not shown) may be implementedwithin automated diagnostic and treatment system 102 and may utilize anerror vector of any device in medical instrument equipment or attachedmedical instrument equipment 108 to create the device usage accuracyscore and utilize the actual patient-measured device data to create themeasurement data accuracy score.

In embodiments, automated diagnostic and treatment system 102 outputsdiagnosis and/or treatment information that may be communicated to thepatient, for example, electronically or in person by a medicalprofessional, e.g., a treatment guideline that may include aprescription for a medication. In embodiments, prescriptions may becommunicated directly to a pharmacy for pick-up or automated homedelivery.

In embodiments, automated diagnostic and treatment system 102 maygenerate an overall health risk profile of the patient and recommendsteps to reduce the risk of overlooking potentially dangerous conditionsor guide the patient to a nearby facility that can treat the potentiallydangerous condition. The health risk profile may assist a treatingdoctor in fulfilling duties to the patient, for example, to carefullyreview and evaluate the patient and, if deemed necessary, refer thepatient to a specialist, initiate further testing, etc. The health riskprofile advantageously reduces the potential for negligence and, thus,medical malpractice.

Automated diagnostic and treatment system 102, in embodiments, comprisesa payment feature that uses patient identification information to accessa database to, e.g., determine whether a patient has previously arrangeda method of payment, and if the database does not indicate a previouslyarranged method of payment, automated diagnostic and treatment system102 may prompt the patient to enter payment information, such asinsurance, bank, or credit card information. Automated diagnostic andtreatment system 102 may determine whether payment information is validand automatically obtain an authorization from the insurance, EHRsystem, and/or the card issuer for payment for a certain amount forservices rendered by the doctor. An invoice may be electronicallypresented to the patient, e.g., upon completion of a consultation, suchthat the patient can authorize payment of the invoice, e.g., via anelectronic signature.

In embodiments, database 103 of patient's information (e.g., a securedcloud-based database) may comprise a security interface (not shown) thatallows secure access to a patient database, for example, by usingpatient identification information to obtain the patient's medicalhistory. The interface may utilize biometric, bar code, or otherelectronically security methods. In embodiments, medical instrumentequipment 108 uses unique identifiers that are used as a control toolfor measurement data. Database 103 may be a repository for any type ofdata created, modified, or received by diagnostic system 100, such asgenerated diagnostic information, information received from patient'swearable electronic devices, remote video/audio data and instructions,e.g., instructions received from a remote location or from theapplication.

In embodiments, fields in the patient's electronic health care record(EHR) are automatically populated based on one or more of questionsasked by diagnostic system 100, measurements taken by the diagnosticsystem 100, diagnosis and treatment codes generated by diagnostic system100, one or more trust scores, and imported patient health care datafrom one or more sources, such as an existing health care database. Itis understood the format of imported patient health care data may beconverted to be compatible with the EHR format of diagnostic system 100.Conversely, exported patient health care data may be converted, e.g., tobe compatible with an external EHR database.

In addition, patient-related data documented by diagnostic system 100provide support for the code decision for the level of exam a doctorperforms. Currently, for billing and reimbursement purposes, doctorshave to choose one of any identified codes (e.g., ICD10 currently holdsapproximately 97,000 medical codes) to identify an illness and providean additional code that identifies the level of physical exam/diagnosisperformed on the patient (e.g., full body physical exam) based on anillness identified by the doctor.

In embodiments, patient answers are used to suggest to the doctor alevel of exam that is supported by the identified illness, e.g., toensure that the doctor does not perform unnecessary in-depth exams forminor illnesses or a treatment that may not be covered by the patient'sinsurance.

In embodiments, upon identifying a diagnostic system 100 generates oneor more recommendations/suggestions/options for a particular treatment.In embodiments, one or more treatment plans are generated that thedoctor may discuss with the patient and decide on a suitable treatment.For example, one treatment plan may be tailored purely foreffectiveness, another one may consider the cost of drugs. Inembodiments, diagnostic system 100 may generate a prescription or labtest request and consider factors, such as recent research results,available drugs and possible drug interactions, the patient's medicalhistory, traits of the patient, family history, and any other factorsthat may affect treatment when providing treatment information. Inembodiments, diagnosis and treatment databases may be continuouslyupdated, e.g., by health care professionals, so that an optimaltreatment may be administered to a particular patient, e.g., a patientidentified as member of a certain risk group.

It is noted that sensors and measurement techniques may beadvantageously combined to perform multiple functions using a reducednumber of sensors. For example, an optical sensor may be used as athermal sensor by utilizing IR technology to measure body temperature.It is further noted that some or all data collected by diagnostic system100 may be processed and analyzed directly within automated diagnosticand treatment system 102 or transmitted to an external reading device(not shown in FIG. 1) for further processing and analysis, e.g., toenable additional diagnostics.

FIG. 2 illustrates an exemplary patient diagnostic measurement systemaccording to embodiments of the present disclosure. As depicted, patientdiagnostic measurement system 200 comprises microcontroller 202,spectrum imaging device, e.g., spectrum imaging device camera 204,monitor 206, patient-medical equipment activity tracking sensors, e.g.,inertial sensor 208, communications controller 210, medical instruments224, identifiable marker, e.g., identifiable marker IR LED 226, powermanagement unit 230, and battery 232. Each component may be coupleddirectly or indirectly by electrical wiring, wirelessly, or optically toany other component in patient diagnostic measurement system 200.Inertial sensor 208 may also be referred to as patient-equipmentactivity sensors inertial sensors 208 or simply sensors 208.

Medical instrument 224 comprises one or more devices that are capable ofmeasuring physical and non-physical characteristics of a patient that,in embodiments, may be customized, e.g., according to varying anatomiesamong patients, irregularities on a patient's skin, and the like. Inembodiments, medical instrument 224 is a combination of diagnosticmedical devices that generate diagnostic data based on patientcharacteristics. Exemplary diagnostic medical devices are heart ratesensors, otoscopes, digital stethoscopes, in-ear thermometers, bloodoxygen sensors, high-definition cameras, spirometers, blood pressuremeters, respiration sensors, skin resistance sensors, glucometers,ultrasound devices, electrocardiographic sensors, body fluid samplecollectors, eye slit lamps, weight scales, and any devices known in theart that may aid in performing a medical diagnosis. In embodiments,patient characteristics and vital signs data may be received from and/orcompared against wearable or implantable monitoring devices that gathersample data, e.g., a fitness device that monitors physical activity.

One or more medical instruments 224 may be removably attachable directlyto a patient's body, e.g., torso, via patches or electrodes that may useadhesion to provide good physical or electrical contact. In embodiments,medical instruments 224, e.g., a contact-less (or non-contact)thermometer, may perform contact-less measurements some distance awayfrom the patient's body. Contact, and non-contact sensors may alsosupport measurements for any number of bioelectric recordings, includingbut not limited to, electrocardiogram (ECG), electroencephalogram (EEG),electromyogram (EMG), electrooculogram (EOG), and electrogastrogram(EGG).

In embodiments, microcontroller 202 may be a secure microcontroller thatsecurely communicates information in encrypted form to ensure privacyand the authenticity of measured data and activity sensor andpatient-equipment proximity information and other information in patientdiagnostic measurement system 200. This may be accomplished by takingadvantage of security features embedded in hardware of microcontroller202 and/or software that enables security features during transit andstorage of sensitive data. Each device in patient diagnostic measurementsystem 200 may have keys that handshake to perform authenticationoperations on a regular basis.

Spectrum imaging device camera 204 is any audio/video device that maycapture patient images and sound at any frequency or image type. Monitor206 is any screen or display device that may be coupled to camera,sensors and/or any part of patient diagnostic measurement system 200.Patient-equipment activity tracking inertial sensor 208 is any single ormulti-dimensional sensor, such as an accelerometer, a multi-axisgyroscope, pressure sensor, and a magnetometer capable of providingposition, motion, pressure on medical equipment or orientation databased on patient interaction. Patient-equipment activity trackinginertial sensor 208 may be attached to (removably or permanently) orembedded into medical instrument 224. Identifiable marker IR LED 226represents any device, heat source, reflective material, proximitybeacon, altimeter, etc., that may be used by microcontroller 202 as anidentifiable marker Like patient-equipment activity tracking inertialsensor 208, identifiable marker IR LED 226 may be attachable to orembedded into medical instrument 224.

In embodiments, communication controller 210 is a wirelesscommunications controller attached either permanently or temporarily tomedical instrument 224 or the patient's body to establish abi-directional wireless communications link and transmit data, e.g.,between sensors and microcontroller 202 using any wireless communicationprotocol known in the art, such as Bluetooth Low Energy, e.g., via anembedded antenna circuit that wirelessly communicates the data. One ofordinary skill in the art will appreciate that electromagnetic fieldsgenerated by such antenna circuit may be of any suitable type. In caseof an RF field, the operating frequency may be located in the ISMfrequency band, e.g., 13.56 MHz. In embodiments, data received bycommunications controller 210 may be forwarded to a host device (notshown) that may run a software application.

In embodiments, power management unit 230 is coupled to microcontroller202 to provide energy to, e.g., microcontroller 202 and communicationcontroller 210. Battery 232 may be a back-up battery for powermanagement unit 230 or a battery in any one of the devices in patientdiagnostic measurement system 200. One of ordinary skill in the art willappreciate that one or more devices in patient diagnostic measurementsystem 200 may be operated from the same power source (e.g., battery232) and perform more than one function at the same or different times.A person of skill in the art will also appreciate that one or morecomponents, e.g., sensors 208, identifiable marker IR LED 226, may beintegrated on a single chip/system, and that additional electronics,such as filtering elements, etc., may be implemented to support thefunctions of medical instrument equipment measurement or usagemonitoring and tracking in patient diagnostic measurement system 200according to the objectives of the invention.

In operation, a patient may use medical instrument 224 to gather patientdata based on physical and non-physical patient characteristics, e.g.,vital signs data, images, sounds, and other information useful in themonitoring and diagnosis of a health-related condition. The patient datais processed by microcontroller 202 and may be stored in a database (notshown). In embodiments, the patient data may be used to establishbaseline data for a patient health profile against which subsequentpatient data may be compared.

In embodiments, patient data may be used to create, modify, or updateEHR data. Gathered medical instrument equipment data, along with anyother patient and sensor data, may be processed directly by patientdiagnostic measurement system 200 or communicated to a remote locationfor analysis, e.g., to diagnose existing and expected health conditionsto benefit from early detection and prevention of acute conditions oraid in the development of novel medical diagnostic methods.

In embodiments, medical instrument 224 is coupled to a number ofsensors, such as patient-equipment tracking inertial sensor 208 and/oridentifiable marker IR LED 226, that may monitor a position/orientationof medical instrument 224 relative to the patient's body when a medicalequipment measurement is taken. In embodiments, sensor data generated bysensor 208, identifiable marker IR LED 226 or other sensors may be usedin connection with, e.g., data generated by spectrum imaging devicecamera 204, proximity sensors, transmitters, bolometers, or receivers toprovide feedback to the patient to aid the patient in properly aligningmedical instrument 224 relative to the patient's body part of interestwhen performing a diagnostic measurement. A person skilled in the artwill appreciate that not all sensors 208, identifiable marker IR LED226, beacon, pressure, altimeter, etc., need to operate at all times.Any number of sensors may be partially or completely disabled, e.g., toconserve energy.

In embodiments, the sensor emitter comprises a light signal emitted byIR LED 226 or any other identifiable marker that may be used as areference signal. In embodiments, the reference signal may be used toidentify a location, e.g., within an image and based on a characteristicthat distinguishes the reference from other parts of the image. Inembodiments, the reference signal is representative of a differencebetween the position of medical instrument 224 and a preferred locationrelative to a patient's body. In embodiments, spectrum imaging devicecamera 204 displays, e.g., via monitor 206, the position of medicalinstrument 224 and the reference signal at the preferred location so asto allow the patient to determine the position of medical instrument 224and adjust the position relative to the preferred location, displayed byspectrum imaging device camera 204.

Spectrum imaging device camera 204, proximity sensor, transmitter,receiver, bolometer, or any other suitable device may be used to locateor track the reference signal, e.g., within the image, relative to abody part of the patient. In embodiments, this augmented reality (AR)method may be accomplished by using an overlay method that overlays animage of a body part of the patient against an ideal model of deviceusage to enable real-time feedback for the patient. The reference signalalong with signals from other sensors, e.g., patient-equipment activityinertial sensor 208, may be used to identify a position, location,angle, orientation, or usage associated with medical instrument 224 tomonitor and guide a patient's placement of medical instrument 224 at atarget location and accurately activate a device for measurement. Inother embodiments, methods of mixed reality, in which users wearing ARgoggles may have virtual objects overlaid on real-world objects (e.g. avirtual clock on a real wall).

In embodiments, e.g., upon receipt of a request signal, microcontroller202 activates one or more medical instruments 224 to performmeasurements and sends data related to the measurement back tomicrocontroller 202. The measured data and other data associated with aphysical condition may be automatically recorded and a usage accuracy ofmedical instrument 224 may be monitored.

In embodiments, microcontroller 202 uses an image in any spectrum,motion signal and/or an orientation signal by patient-equipment activityinertial sensor 208 to compensate or correct the vital signs data outputby medical instrument 224. Data compensation or correction may comprisefiltering out certain data as likely being corrupted by parasiticeffects and erroneous readings that result from medical instrument 224being exposed to unwanted movements caused by perturbations or, e.g.,the effect of movements of the patient's target measurement body part.

In embodiments, signals from two or more medical instruments 224, orfrom medical instrument 224 and patient-activity activity systeminertial sensor 208, are combined, for example, to reduce signal latencyand increase correlation between signals to further improve the abilityof patient diagnostic measurement system 200 to reject motion artifactsto remove false readings and, therefore, enable a more accurateinterpretation of the measured vital signs data.

In embodiments, spectrum imaging device camera 204 and medicalinstrument monitor 206 displays actual or simulated images and videos ofthe patient and medical instrument 224 to assist the patient in locatinga desired position for medical instrument 224 when performing themeasurement so as to increase measurement accuracy. Spectrum imagingdevice camera 204 may use image or facial recognition software toidentify and display eyes, mouth, nose, ears, torso, or any other partof the patient's body as reference.

In embodiments, patient diagnostic measurement system 200 uses machinevision software that analyzes measured image data and compares imagefeatures to features in a database, e.g., to detect an incomplete imagefor a target body part, to monitor the accuracy of a measurement anddetermine a corresponding score. In embodiments, if the score fallsbelow a certain threshold, patient diagnostic measurement system 200 mayprovide detailed guidance for improving measurement accuracy or toreceive a more complete image, e.g., by providing instructions on how tochange an angle or depth of an otoscope relative to the patient's ear.

In embodiments, the deep learning system may use an overlay method tomimic a patient's posture/movements to provide detailed and interactiveinstructions, e.g., by displaying a character, image of the patient,graphic, or avatar on monitor 206 to provide feedback to the patient.The instructions, image, or avatar may start or stop and decide whathelp instruction to display based on the type of medical instrument 224,the data from spectrum imaging device camera 204, patient-equipmentactivity sensors inertial sensors 208, bolometer, transmitter andreceiver, and/or identifiable marker IR LED 226 (an image, a measuredposition or angle, etc.), and a comparison of the data to idealizeddata. This further aids the patient in correctly positioning andoperating medical instrument 224 relative to the patient's body, ensuresa high level of accuracy when operating medical instrument 224, andsolves potential issues that the patient may encounter when usingmedical instrument 224.

In embodiments, instructions may be provided via monitor 206 anddescribe in audio/visual format and in any desired level of detail, howto use medical instrument 224 to perform a diagnostic test ormeasurement, e.g., how to take temperature, so as to enable patients toperform measurements of clinical-grade accuracy. In embodiments, eachsensor 208, identifiable marker IR LED 226, e.g., proximity, bolometer,transmitter/receiver may be associated with a device usage accuracyscore. A device usage accuracy score generator (not shown), which may beimplemented in microcontroller 202, may use the sensor data to generatea medical instrument usage accuracy score that is representative of thereliability of medical instrument 224 measurement on the patient. Inembodiments, the score may be based on a difference between an actualposition of medical instrument 224 and a preferred position. Inaddition, the score may be based on detecting a motion, e.g., during ameasurement. In embodiments, the device usage accuracy score is derivedfrom an error vector generated for one or more sensors 208, identifiablemarker IR LED 226, deep learning system 105. In embodiments, the deviceusage accuracy score is derived from an error vector generated for oneor more sensors 208, identifiable marker IR LED 226. The resultingdevice usage accuracy score may be used when generating or evaluatingmedical diagnosis data.

In embodiments, microcontroller 202 analyzes the patient measuredmedical instrument data to generate a trust score indicative of theacceptable range of the medical instrument. For example, by comparingthe medical instrument measurement data against reference measurementdata or reference measurement data that would be expected from medicalinstrument 224. As with device usage accuracy score, the trust score maybe used when generating or evaluating a medical diagnosis data.

FIG. 3A illustrates an exemplary system 300 for measuring a patient'sbody parts utilizing audio medical instruments and deep learning systemsaccording to embodiments of the present disclosure. System 300 maycomprise automated diagnostic and treatment system 102, deep learningsystem 105, doctor interface station 104 and patient kiosk 301. Patientkiosk 301 may comprise patient interface station 106, kiosk camerasincluding sensors/camera-1 303, sensors/camera-2 304 and instrument 312.As illustrated, patient 310 is present in patient kiosk 301. Kioskcameras may also be referred to as camera modules. Instrument 312 maybe, but not limited to, a stethoscope. Deep learning system 105 may be,but not limited to, a deep convolution neural network (DCNN). System 300facilities a method of computer vision which may utilize pose estimationand object tracking. A stethoscope may be operable to detect an audiosignal of the patient when positioned within an object tracking area ofthe patient.

An objective of system 300 includes the accurate acquisition of medicalmeasurement data of a target spot of a body part of patient 310. Toassist in acquiring accurate medical measurement data, the automateddiagnostic and treatment system 102 provides instructions to patient 310to allow patient 310 to precisely position instrument 312 in proximityto a target spot of a body part of patient 310. That is, the movement ofinstrument 312 may be controlled by the patient 310. Based on a seriesof images acquired from the kiosk cameras and instrument 312, theseinstructions may be generated. Subsequent images from instrument 312 areanalyzed by automated diagnostic and treatment system 102 utilizingdatabase 103 and deep learning system 105 to obtain measured medicaldata. The accuracy of the instrument 312 positioning and the medicalmeasured data may be determined by automated diagnostic and treatmentsystem 102, and deep learning system 105 via an error thresholdmeasurement. When accurately positioned, quality medical measurementsare generated for the target spot of a body part of patient 310. Asillustrated in FIG. 3A, patient 310 has positioned instrument 312, whichmay be a stethoscope, in proximity to the heart of patient 310. If thestethoscope is within a recording threshold of the heart of patient 310,a recording may be obtained. The recording threshold may be, but withoutlimitations, less than one inch. In other embodiments, instrument 312may be positioned within a recording threshold to another body part. Inother embodiments, patient 310 may be fitted with haptic/tactilefeedback sensors which may augment the patient guidance instructions fortaking measurements. These haptic feedback devices, or haptic feedbacksensors, may include, but are not limited to, vibration motors (similarto those found in smart phones) fitted around the medical instrumentequipment to give tactile instructions (e.g. vibration on the left sideof the medical device to instruct the patient to move the medical deviceleft), or flywheels to give kinesthetic feedback such as resistingmovement in undesirable directions, and aiding movement in the correctdirection (similar to certain virtual reality peripherals). Instrument312 may be wirelessly coupled to patient interface station 106.

The automated diagnostic and treatment system 102, deep learning system105, doctor interface station 104 and patient interface station 106 areoperable as previously described herein relative to FIG. 1.Collectively, these systems and stations, using pose estimation andobject tracking methods, may determine with computer vision if thestethoscope has been moved to the location by patient 310 to allow anaudio recording. The automated diagnostic and treatment system 102 maysend a command to the patient interface station 106 requestingsensors/camera-1 303 and sensors/camera-2 304 to capture images ofpatient 310 and instrument 312. The captured images (sometimes called“posed” images) may be analyzed by the deep learning system 105 todetermine the position of instrument 312 relative to the body part. Fromthis positioning analysis, the automated diagnostic and treatment system102 may provide an initial set of instructions to patient 310 to assistpatient 310 to position instrument 312 within a pose threshold to thetarget spot of the body part. After patient 310 acts on the first set ofinstructions, the deep learning system 105 may captures a second set ofimages and determines if instrument 312 is within a the pose thresholdof the target spot of the body part. The pose threshold may be less thanone inch.

As illustrated in patient kiosk 301, there are two camera/sensormodules. In some embodiments, there may be N camera/sensor modules,where N is equal to one of more camera/sensor modules.

A stethoscope examination by the automated diagnostic and treatmentsystem 102 and deep learning system 105 may include the following tasks:

After a pose estimate, using pose estimation and object trackingmethods, the deep learning system 105 may determine, with computervision, if instrument 312 (stethoscope) has been moved to a correctlocation by the patient 310 for an audio recording. A correct locationmeans the stethoscope is positioned within an object tracking area ofthe body, or within a recording threshold of a recording site.

Using digital signal processing techniques (DSP), the deep learningsystem 105 may confirm that the stethoscope has been turned on.

Using DSP techniques, the deep learning system 105 may “clean” the audiosignal acquired from stethoscope. This step may be referred to aspre-processing.

Using Machine Learning (ML) techniques, the deep learning system 105 maydetermine if a Signal of Interest (SoI), in this case, e.g., but withoutlimitations, heart, lung, bowel, muscle, skin or abdomen sounds, arepresent in the pre-processed audio sample. This step may be referred toas “validation”. The ML techniques may be implemented in the automateddiagnostic and treatment system 102.

If the SoI is present in the pre-processed audio sample, the deeplearning system 105 may use ML techniques to evaluate characteristics inthe signal which indicate the presence of abnormalities in the organbeing measure, e.g. crackling of lungs. This step may be referred to as“classification” or “symptom classification”.

The aforementioned tasks will be further discussed relative to FIGS. 4A,4B, 4C, 4D, 4E and FIGS. 5A, 5B, 5C.

FIG. 3B illustrates another exemplary system 320 for measuring apatient's body parts utilizing cameras and sensors according toembodiments of the present disclosure. System 320 is based on theprinciple of determine the time-of-flight (TOF) for each emitted pulseof light. Light detection and ranging systems, such as camera/sensors322, may employ pulses of light to measure distance to an object basedon the time of flight (TOF) of each pulse of light. A pulse of lightemitted from a light source of a light detection and ranging systeminteracts with a distal object. A portion of the light reflects from theobject and returns to a detector of the light detection and rangingsystem. Based on the time elapsed between emission of the pulse of lightand detection of the returned pulse of light, the distance to the objectmay be estimated.

The light pulse may hit multiple objects, each having a differentdistance from the laser, causing multi-return signals to be received bythe light detection and ranging system detector. Multi-return signalsmay provide more information of the environment to improve mapping orreconstruction. A dedicated detector may be required to preciselyidentify each return with its associated time delay information. Theresulting images may be referred to as “pose images”. In other words,one or more camera modules may provide a pose estimate based on time offlight of multiple light signals emitted from each of the one or morecamera modules and reflected back to the one or more camera modules.

As illustrated in FIG. 3B, camera/sensors 322 may emits signals S1, S2and S3. These signals may reflect off patient 330 or off a fixedsurface, such as a wall or floor. In the kiosk, the walls and floor mayhave identifiers that can assist in the image identification of the bodypart of patient 330. The reflection of signals S1, S2 and S3 may bedetected by camera/sensors 322 and communicated to a diagnostic system,such as automated diagnostic and treatment system 102. Automateddiagnostic and treatment system 102 can determine the distancesassociated with signals S1, S2 and S3 based on the TOF of each signal.This process may result in a pose estimate, which may be viewed as apose image. Identifier 324 may assist in identifying a position of apatient's body parts or a medical instrument.

FIGS. 4A, 4B, 4C, and 4D illustrates an exemplary method for identifyinga patient's body parts and body recording sites according to embodimentsof the present disclosure. FIG. 4A illustrates embodiment 400 of patient402. To begin, four coordinate points ((x₁, y₁), (x₂, y₂), (x₄, y₄),(x₃, y₃)) of a bounding box for the body trunk may be estimatedutilizing a deep learning algorithm from deep learning system 105, asillustrated in FIG. 4A. Indicator 404 identifies one potential recordingsite, as noted by the circle, which has an image like a poker chip.Embodiment 400 comprises a total of the five potential recording sitesin the bounding box. Next, in embodiment 420 of FIG. 4B, the boundingbox may be subdivided into four quadrants. The four quadrants areidentified by the intervals of L₁ and L₂ on FIG. 4B. Then, the fourquadrants may be further sub-divided into a grid, as illustrated inembodiment 440 of FIG. 4C. In embodiment 440, the grid is based onsegments having values of ⅓ (L₁) and (L₂)/4. Embodiment 460 of FIG. 4Dmay identify certain grid areas corresponding to potential recordingsites, s₁, s₂, s₃, s₄, and s₅. This grid identification process may besupported by patient kiosk 301, automated diagnostic and treatmentsystem 102 and deep learning system 105. One skilled in the art mayrecognize that other exemplary methods may be utilized for identifying apatient's body parts and body recording sites.

FIG. 4E illustrates an exemplary embodiment 480 for the identificationof body parts of patient 402 and the positioning of a stethoscope forrecording audio measurement of the body according to embodiments of thepresent disclosure. More specifically, FIG. 4E illustrates the use of MLand deep learning system 105 to obtain pose estimates of the body parts.With the pose estimates, which are generated via from computer vision,deep learning system 105, joints 406 and limbs 408 may be identified.The white dots on FIG. 4E illustrate the joints including LShoulder,Rshoulder, RElbow, RWrist, RHip, LHip, LElbow and LWrist. The gray dotson FIG. 4E illustrate potential recording sites, as indicated by thenumbers 1, 2, 3 and 0. The lines between the white dots illustrate thelimbs 408 of the patient 402. Eight joints are identified in FIG. 4E.Using a complementary object tracking deep learning algorithm, thealgorithm tracks the identifiable marker (e.g. hand, stethoscope, IRLED) location and determines if the hand is holding the stethoscope hasmoved into the object tracking area 412. Object tracking area 412 maycomprise at least one of the recording sites. When a stethoscope moveswithin object tracking area 412, a stethoscope may be able to record anaudio signal. Effectively, when a stethoscope moves within objecttracking area 412, the stethoscope is within a recording threshold of arecording site. One skilled in the art may recognize that other methodsand systems may be utilized for the identification of a patient's bodyparts and stethoscope positioning for recording audio measurements ofthe body.

In some embodiments, there may be assumptions related to the handlingand position of the stethoscope in the hand of patient 402. Theassumptions may include: 1) When held, the stethoscope will be occludedby the hand. This means the deep learning system 105 cannot reliableperform object tracking on the stethoscope. 2) The hand which holds thestethoscope may be a viable proxy for the stethoscope location. 3) Whentracking the hand holding of the stethoscope, the position of thestethoscope may be estimated with an acceptable level of accuracy. Thehand in this case can be referred to as the Identifiable Marker. 4) Asensor board attached to the stethoscope can validate that thestethoscope is in the hand of patient 402 by sending a signal to theautomated diagnostic and treatment system 102. 5) The validation thatthe stethoscope is positioned in the correct location for recording maybe based on computer vision and the presence of the signal of interest(SoI) in a stethoscope signal.

FIGS. 5A and 5B are flowcharts 500 and 540 of exemplary methods formaking accurate medical instrument patient audio measurements, accordingto embodiments of the present disclosure. This method references astethoscope, but those skilled in the art may recognize that the methodmay apply to other types of audio instruments. The method may comprisethe following steps:

Determining with computer vision the stethoscope is positioned within apose threshold to recording sites of body parts of the patient 310.Computer vision generates pose estimates from the kiosk cameras. Poseestimation and object tracking methods may identify multiple recordingsites on the patient 310. (step 502)

Implementing a RMS check or confidence check. Using digital signalprocessing techniques (DSP) implemented by the deep learning system 105,confirm whether the stethoscope operating, i.e., is turned on. Morespecifically, to ensure the user has turned the stethoscope on, DSPtechniques may be utilized by the evaluation of the incoming audiosignal from the stethoscope by analyzing the Root Mean Square (RMS) ofthe amplitude of the incoming audio signal. (step 504) If no signal ispresent (or the RMS value is too low), patient kiosk 301 may instructthe user to turn on the stethoscope, or increase the volume. (step 505),then repeat step 504.

If the stethoscope is turned on, providing, by the automated diagnosticand treatment system 102, instruction to patient 310 to move thestethoscope to a recording site of body part of patient. Theinstructions provide the user with a set of visual directions and/orhaptic/tactile feedback to move the stethoscope into the correctlocation for the first recording site. Recording sites may be identifiedby an object tracking area 412 (step 506)

Is stethoscope positioned at a recording site for a body part ofpatient? If yes, the position of the tracked object (e.g. hand of thepatient, or medical device) is validated. Each incoming frame/image fromthe patient kiosk 301 may be evaluated by the deep learning system 105to determine if the patient has moved the stethoscope within a recordingthreshold of a recording site. The recording site may be within anobject tracking area, as previously discussed. This determination may bebased on the corroborating evidence from the object tracking algorithmidentifying the location of the hand at a recording site, poseestimation of the patient, and incoming data from the sensor boardattached to the stethoscope. If the stethoscope is within a recordingthreshold of a recording site, the stethoscope may record an audiosignal of the patient. For example, has the identifiable marker (i.e.hand) moved to a correct recording site location? (step 508)

If the stethoscope has not moved within a recording threshold of arecording site body part of patient, repeat step 506. That is, provideadditional instructions to the patient to move the stethoscope within arecording threshold of the one of the one or more recording sites of thepatient.

If the stethoscope has moved and is positioned at a recording site of abody part of patient, a signal may be recorded based on an audio signalfrom the stethoscope. Subsequently, the recorded audio signal may bepre-processed to “clean” or filter the audio signal. DSP techniques maybe utilized to “clean” the audio signal acquired from stethoscope. Thisprocess may be referred to as “pre-processing” or “DSP clean-up” andresults in a pre-processed recorded audio signal or pre-processed audiosample. The sample recording may be 5-10 second in length, or as long asrequired. (step 510)

Is a Signal of Interest (SoI) present in the recorded audio signal?Using Machine Learning (ML) techniques, determine if the Signal ofInterest (SoI) is present in the pre-processed audio sample. SoI may bebased on heart, lung, bowel, muscle, or skin audio sounds, This step maybe referred to as “validation” (step 512)

If the SoI is present, the deep learning system 105 will apply MLtechniques to determine if there are any characteristic soundsindicative of a potential ailment or abnormality in the organ beingmeasured, e.g. crackling of lungs. This step may be referred to as“classification” or “symptom classification. (step 514)

Then, repeating steps 502, 504, 506, 510, 512, and 514 for theother/remainder of the recording sites on the body parts of patient 310.That is, repeating, by the diagnostic system, the determination ofwhether the signal of interest is present in pre-processed audio samplesfor a remainder of the one or more recording sites on the body parts ofpatient 310. (step 516)

Analyzing the results and storing results in a deep learning database.Determine if additional measurements of body parts of patient 310 arerecommended. (step 518)

If the SoI is not present, instructions may be given to the patient toadjust the location of the stethoscope or to adjust a position of therecording device on the recording site to ensure adequate transductionof a physical phenomenon being recorded by the recording device, e.g. topress a stethoscope more firmly into the recording site. (step 513)Then, repeating steps 506, 508, 510, 512 and 514.

FIG. 5C is a flow chart 560 of an illustrative exemplary method fordetermining where recording locations are on a patient's body accordingto embodiments of the present disclosure. Utilizing a computer visionand deep learning algorithms, the method may comprise the followingsteps:

Identifying key joint location, and key feature location. That is,identifying points of key joints and limbs. (Step 561)

Using shoulder and hip key joint locations (x₁,y₁), (x₂,y₂), (x₃, y₃),(x₄, y₄), to create a bounding box for the trunk of the patient. SeeFIG. 4A. The bounding box may be the upper half of the torso of a bodyor the trunk of the body. In other words, the points of key joints andlimbs may be used to create a bounding box. This step may be implementedwith pose estimation. (Step 562)

Subdividing the bounding box into four quadrants. (see FIG. 4B), (step564)

Further subdividing the four quadrants into a grid. (see FIG. 4C) (step566)

Identifying certain grid areas corresponding to potential recordingsites, s₁, s₂, s₃, s₄, and s₅ (see FIG. 4D) (step 568)

Identifying/mapping an object tracking area comprising at least onerecording site. (step 572)

Determining if the patient has moved the stethoscope to the objecttracking area and to the at least one recording site. (step 574)

If yes, the method continues at step 510: Recording a signal from thestethoscope and pre-processing the audio signal. (DSP clean up).Subsequently, the method continues with step 512: Is a Signal ofInterest (SoI) present in the recorded signal? Using Machine Learning(ML) techniques, determine if the Signal of Interest (SoI) is present inthe pre-processed audio sample. SoI may be based on heart, lung, bowel,muscle, or skin audio sounds, This step may be referred to as“validation”.

If no, use pose estimation, and object tracking to determine if thepatient has moved the stethoscope to the object tracking area and to theat least one recording site, then repeat step 506. (step 576)

In embodiments, a time may be calculated at which the selected treatmentis expected to show a result and patient feedback may be requested,e.g., as part of a feedback process, to improve diagnosis and treatmentreliability. In embodiments, the selected treatment and/or the patientfeedback may be used to adjusting one or more of the metrics. Forexample, based on one or more of the metrics a series of treatment plansmay be generated by using an algorithm that combines the metrics. Inembodiments, calculating the selected treatment comprises using cost asa factor.

One skilled in the art will recognize that: (1) certain steps mayoptionally be performed; (2) steps may not be limited to the specificorder set forth herein; and (3) certain steps may be performed indifferent orders; and (4) certain steps may be done concurrently.

In embodiments, one or more computing systems, such asmobile/tablet/computer or the automated diagnostic system, may beconfigured to perform one or more of the methods, functions, and/oroperations presented herein. Systems that implement at least one or moreof the methods, functions, and/or operations described herein maycomprise an application or applications operating on at least onecomputing system. The computing system may comprise one or morecomputers and one or more databases. The computer system may be a singlesystem, a distributed system, a cloud-based computer system, or acombination thereof.

It shall be noted that the present disclosure may be implemented in anyinstruction-execution/computing device or system capable of processingdata, including, without limitation phones, laptop computers, desktopcomputers, and servers. The present disclosure may also be implementedinto other computing devices and systems. Furthermore, aspects of thepresent disclosure may be implemented in a wide variety of waysincluding software (including firmware), hardware, or combinationsthereof. For example, the functions to practice various aspects of thepresent disclosure may be performed by components that are implementedin a wide variety of ways including discrete logic components, one ormore application specific integrated circuits (ASICs), and/orprogram-controlled processors. It shall be noted that the manner inwhich these items are implemented is not critical to the presentdisclosure.

Having described the details of the disclosure, an exemplary system thatmay be used to implement one or more aspects of the present disclosureis described next with reference to FIG. 6. Each of patient interfacestation 106 and automated diagnostic and treatment system 102 in FIG. 1may comprise one or more components in the system 600. As illustrated inFIG. 6, system 600 includes a central processing unit (CPU) 601 thatprovides computing resources and controls the computer. CPU 601 may beimplemented with a microprocessor or the like, and may also include agraphics processor and/or a floating point coprocessor for mathematicalcomputations. System 600 may also include a system memory 602, which maybe in the form of random-access memory (RAM) and read-only memory (ROM).

A number of controllers and peripheral devices may also be provided, asshown in FIG. 6. An input controller 603 represents an interface tovarious input device(s) 604, such as a keyboard, mouse, or stylus. Theremay also be a scanner controller 605, which communicates with a scanner606. System 600 may also include a storage controller 607 forinterfacing with one or more storage devices 608 each of which includesa storage medium such as magnetic tape or disk, or an optical mediumthat might be used to record programs of instructions for operatingsystems, utilities and applications which may include embodiments ofprograms that implement various aspects of the present disclosure.Storage device(s) 608 may also be used to store processed data or datato be processed in accordance with the disclosure. System 600 may alsoinclude a display controller 609 for providing an interface to a displaydevice 611, which may be a cathode ray tube (CRT), a thin filmtransistor (TFT) display, or other type of display. System 600 may alsoinclude a printer controller 612 for communicating with a printer 613. Acommunications controller 614 may interface with one or morecommunication devices 615, which enables system 600 to connect to remotedevices through any of a variety of networks including the Internet, anEthernet cloud, an FCoE/DCB cloud, a local area network (LAN), a widearea network (WAN), a storage area network (SAN) or through any suitableelectromagnetic carrier signals including infrared signals.

In the illustrated system, all major system components may connect to abus 616, which may represent more than one physical bus. However,various system components may or may not be in physical proximity to oneanother. For example, input data and/or output data may be remotelytransmitted from one physical location to another. In addition, programsthat implement various aspects of this disclosure may be accessed from aremote location (e.g., a server) over a network. Such data and/orprograms may be conveyed through any of a variety of machine-readablemedium including, but are not limited to: magnetic media such as harddisks, floppy disks, and magnetic tape; optical media such as CD-ROMsand holographic devices; magneto-optical media; and hardware devicesthat are specially configured to store or to store and execute programcode, such as application specific integrated circuits (ASICs),programmable logic devices (PLDs), flash memory devices, and ROM and RAMdevices.

Embodiments of the present disclosure may be encoded upon one or morenon-transitory computer-readable media with instructions for one or moreprocessors or processing units to cause steps to be performed. It shallbe noted that the one or more non-transitory computer-readable mediashall include volatile and non-volatile memory. It shall be noted thatalternative implementations are possible, including a hardwareimplementation or a software/hardware implementation.Hardware-implemented functions may be realized using ASIC(s),programmable arrays, digital signal processing circuitry, or the like.Accordingly, the “means” terms in any claims are intended to cover bothsoftware and hardware implementations. Similarly, the term“computer-readable medium or media” as used herein includes softwareand/or hardware having a program of instructions embodied thereon, or acombination thereof. With these implementation alternatives in mind, itis to be understood that the figures and accompanying descriptionprovide the functional information one skilled in the art would requireto write program code (i.e., software) and/or to fabricate circuits(i.e., hardware) to perform the processing required.

It shall be noted that embodiments of the present disclosure may furtherrelate to computer products with a non-transitory, tangiblecomputer-readable medium that have computer code thereon for performingvarious computer-implemented operations. The media and computer code maybe those specially designed and constructed for the purposes of thepresent disclosure, or they may be of the kind known or available tothose having skill in the relevant arts. Examples of tangiblecomputer-readable media include, but are not limited to: magnetic mediasuch as hard disks, floppy disks, and magnetic tape; optical media suchas CD-ROMs and holographic devices; magneto-optical media; and hardwaredevices that are specially configured to store or to store and executeprogram code, such as application specific integrated circuits (ASICs),programmable logic devices (PLDs), flash memory devices, and ROM and RAMdevices. Examples of computer code include machine code, such asproduced by a compiler, and files containing higher level code that areexecuted by a computer using an interpreter. Embodiments of the presentdisclosure may be implemented in whole or in part as machine-executableinstructions that may be in program modules that are executed by aprocessing device. Examples of program modules include libraries,programs, routines, objects, components, and data structures. Indistributed computing environments, program modules may be physicallylocated in settings that are local, remote, or both.

For purposes of this disclosure, an information handling system mayinclude any instrumentality or aggregate of instrumentalities operableto compute, calculate, determine, classify, process, transmit, receive,retrieve, originate, switch, store, display, communicate, manifest,detect, record, reproduce, handle, or utilize any form of information,intelligence, or data for business, scientific, control, or otherpurposes. For example, an information handling system may be a personalcomputer (e.g., desktop or laptop), tablet computer, mobile device(e.g., personal digital assistant (PDA) or smart phone), server (e.g.,blade server or rack server), a network storage device, or any othersuitable device and may vary in size, shape, performance, functionality,and price. The information handling system may include random accessmemory (RAM), one or more processing resources such as a centralprocessing unit (CPU) or hardware or software control logic, ROM, and/orother types of nonvolatile memory. Additional components of theinformation handling system may include one or more disk drives, one ormore network ports for communicating with external devices as well asvarious input and output (I/O) devices, such as a keyboard, a mouse,touchscreen and/or a video display. The information handling system mayalso include one or more buses operable to transmit communicationsbetween the various hardware components.

One skilled in the art will recognize no computing system or programminglanguage is critical to the practice of the present disclosure. Oneskilled in the art will also recognize that a number of the elementsdescribed above may be physically and/or functionally separated intosub-modules or combined together.

It will be appreciated to those skilled in the art that the precedingexamples and embodiment are exemplary and not limiting to the scope ofthe present disclosure. It is intended that all permutations,enhancements, equivalents, combinations, and improvements thereto thatare apparent to those skilled in the art upon a reading of thespecification and a study of the drawings are included within the truespirit and scope of the present disclosure.

What is claimed is:
 1. A method comprising: determining, by a diagnosticsystem, a recording device is positioned within a pose threshold to oneor more recording sites of body parts of a patient; confirming, by thediagnostic system, the recording device is turned on; providing, by thediagnostic system, instructions to the patient to move the recordingdevice to one of the one or more recording sites; recording, by thediagnostic system, an audio signal at the one of the one or morerecording sites when the recording device is positioned within arecording threshold to the one of the one or more recording sites;pre-processing, by the diagnostic system, the recorded audio signal, bythe diagnostic system, using digital signal processing techniques, andgenerating a pre-processed audio sample; and determining, by thediagnostic system, using machine learning, when a signal of interest ispresent in the pre-processed audio sample, wherein the diagnostic systemcomprises a deep learning system and diagnostic system camera/sensors.2. The method according to claim 1, further comprising: in response tonot confirming that the recording device is turned on, providinginstructions, by the diagnostic system, to the patient to turn on therecording device.
 3. The method according to claim 1, wherein to confirmthat the recording device is turned on, using machine learning toevaluate an incoming audio signal from the recording device by analyzingan amplitude of the incoming audio signal.
 4. The method according toclaim 1, further comprising: in response to determining that therecording device is not within the recording threshold of the one of theone or more recording sites, providing, by the diagnostic system,additional instructions to the patient to move the recording devicewithin the recording threshold of the one of the one or more recordingsites.
 5. The method according to claim 1, wherein the deep learningsystem evaluates images received from the diagnostic systemcamera/sensors to determine visually if the recording device has movedwithin the recording threshold of the recording site by object trackingand evaluating posture and pose of the patient, and an identifiablemarker.
 6. The method according to claim 1, further comprising: inresponse to determining that the signal of interest is present in thepre-processed audio sample, the deep learning system, using machinelearning, evaluates characteristics in the pre-processed audio samplewhich indicate presence of abnormalities in an organ or a structurebeing measured.
 7. The method according to claim 6, wherein the organbeing measured is a heart, lung, bowel, muscle, skin or abdomen.
 8. Themethod according to claim 1, further comprising: in response todetermining that the signal of interest is not present in thepre-processed audio sample, providing: (i) additional instructions tothe patient to move the recording device within the recording thresholdof the one of the one or more recording sites, and/or (ii) additionalinstruction to the patient to adjust a position of the recording deviceon the recording site to ensure adequate transduction of a physicalphenomenon being recorded by the recording device.
 9. The methodaccording to claim 1, further comprising: repeating, by the diagnosticsystem, the determination of whether the signal of interest is presentin pre-processed audio samples for a remainder of the one or morerecording sites.
 10. The method according to claim 1, wherein thediagnostic system positions the recording device within the recordingthreshold to the one of the one or more recording sites by: identifyingpoints of key joints and limbs; using the points of key joints and limbsto create a bounding box, and mapping an object tracking area on a gridof the bounding box.